The new window shouldn't matter. If the user can't log in at the new log in screen then there's some other ACL issue at work. If they can log in, then it sounds like is that you are hitting a different hostname in the new window.
For example, if a user logs in on
http://hostname1.yourdomain.com/db.nsf and authenticates there, the authentication cookie is stored on the browser for the remainder of that session. The cookie is bound to that hostname.
If the user is redirected to
http://hostname2.yourdomain.com/db.nsf they will need to authenticate again, even if it is the same physical server and database as before.